System and method for estimating rig state using computer vision for time and motion studies

ABSTRACT

The invention relates to a system and method for estimating global rig state. The system comprises a model incorporating multiple variables related to rig state, at least one camera operably connected to at least one processor wherein said camera is capable of gathering visual data regarding at least one variable of rig state and said processor is capable of compiling rig state data, estimating global rig state, or both. The system further comprises multiple sensors for measuring variables related to global rig state wherein said sensors are operably connected to said processor. The method comprises sensing various aspects of the rig state, collecting visual data corresponding with said sensor data, compiling multiple sources of rig data, and estimating the overall rig state.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/078,587 filed Nov. 12, 2014. Applicant incorporates by referenceherein application Ser. No. 62/078,587 in its entirety.

FIELD OF THE INVENTION

The invention relates to systems and methods for estimating rig statesusing computer vision for time and motion studies.

BACKGROUND AND SUMMARY

Modern drilling involves scores of people and multiple inter-connectingactivities. Obtaining real-time information about ongoing operations isof paramount importance for safe, efficient drilling. As a result,modern rigs often have thousands of sensors actively measuring numerousparameters related to vessel operation, in addition to information aboutthe down-hole drilling environment.

Despite the multitude of sensors on today's rigs, a significant portionof rig activities and sensing problems remain difficult to measure withclassical instrumentation, and person-in-the-loop sensing is oftenutilized in place of automated sensing.

By applying automated, computer-based video interpretation, continuous,robust, and accurate assessment of many different phenomena can beachieved through pre-existing video data without requiring aperson-in-the-loop. Automated interpretation of video data is known ascomputer vision, and recent advances in computer vision technologieshave led to significantly improved performance across a wide range ofvideo-based sensing tasks. Computer vision can be used to improvesafety, reduce costs and improve efficiency.

There are hundreds of sensors on a modern drilling rig, each of whichrecords detailed information about a very small part of the rigbehavior—e.g., one sensor measures the flow in a pipe, another measuresdown-bore pressure, another measures pit volume. Making sense of all thedata from each individual sensor is typically left to the rig operator,or, in some cases, to an automated (computerized) process that isresponsible for alerting the user to unexpected measurements orwell-behaviors (e.g., losses, influxes, stuck pipe, etc.).

Due to the complexity of modern drilling and the number of possibleinteracting human behaviors on a rig, aggregating data from each ofthese individual sensors is a very complicated task. Importantly, noneof the individual sensors has direct access to the big-picture rigstate. In other words, the large-scale activities occurring on the rig(e.g., adding barite to the pits, tripping, drilling ahead) are notreadily deducible from each sensors readings in isolation, nor inaggregate.

Although not directly observable in the low-level sensor data, therig-state is often obvious to the drilling engineer, who can visuallysurvey the rig, and identify large-scale behaviors. For example, the rigoperator could visually deduce that “we are pulling out of the rig, butcurrently the drill stand is in slips,” or “someone is adding barite tothe pits, and is doing so after the flow sensor, which will cause pitvolume measurements and flow-out measurements to diverge, so I shouldpay attention to the flow-out measurements to identify possible influxor losses, but ignore changes in the pit volume for the time being.”This level of visual information aggregation provides significantbenefits to human-in-the-loop data processing, and also provides a levelof information not currently easily available in post-drilling analyses.

Significant effort is often spent analyzing hours and days of videofeeds from rigs to perform time & motion studies (“T&M studies”), whichare typically focused on one or two classical “drilling states”—e.g.,improving time spent pulling out of hole. Since raw video data is notannotated, it is extremely difficult to “jump” to the next instance ofany particular state without having watched and annotated the wholevideo first; which is very time consuming and error-prone. It is alsocurrently not possible to automatically determine, from the observedvideo and down-well data, the amount of time spent in any one state.However, these important T&M parameters could be automatically estimatedif state data could be automatically inferred and added to the videodata stream. Incorporation of this information could make all other rigautomation and sensing technologies operate more smoothly and produceoverall higher quality results.

Therefore there is a need for an automated computer vision basedtechnique for observing and estimating large-scale information about thecomplete rig state. This will enable improved low-level processing byincorporating rig state into decision making, and enable improved T&Manalyses and processing by providing automatically annotated video feedsfor rig understanding.

Furthermore, the incorporation of computer vision technologies toestimate rig state enables automated identification of dangerous oruncommon states and state transitions. Alerts regarding uncommonscenarios may be presented to the user in plain-English descriptions ofthe observed state.

Finally, automated rig state detection also enables improved automateddecision making and rig control.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example of classical well state transitions.

FIG. 2 shows one embodiment of a more detailed representation of theunderlying rig state.

FIG. 3 depicts a low-level state-tracking as part of a larger stateestimation process.

FIG. 4 shows one of many embodiments of the disclosed rig stateestimation system utilizing multiple cameras.

FIG. 5 shows the steps of one of many methods for estimating rig state.

DETAILED DESCRIPTION

The Rig State Estimation system, (“RSE”), may contain several partsincluding a model of rig state not as a single discrete “state” that canbe entered and exited from, but as a collection of facts about a rigthat may be estimated and updated in real-time. In addition to multiplesensors 120, which may provide information such as measured surfacereadings or down-well measurements, RSE uses computer vision and machinelearning algorithms specifically designed to estimate the rig state fromvideo cameras 102. The resulting information about rig state may then beincorporated into a display, along with other relevant pieces ofinformation (e.g., video frames) and the state information is used toimprove algorithm performance by altering the relative importance of thevariables under consideration and other algorithm-level modifications.The information may also be used to form T&M databases 112 and reports,and to annotate the collected video with relevant T&M information. Thiswould allow, for example, someone to immediately jump to all theinstances of “roughneck engaged” or “adding barite to the pits” in avideo feed. The information can help identify and flag uncommon events,presenting information about these events to the end-user in plainEnglish. It may also be used to automate and generally improve rigbehaviors. As a result total rig performance may be significantlyenhanced.

Classical estimates of rig “state” utilize a finite-set of discretestates corresponding to large-scale rig behavior, e.g., “Drilling”,“Pulling Out Of Hole”, “Out Of Hole”, “Running Into Hole”, etc. This isvisualized in FIG. 1. While these kinds of state models are useful whenthe number of states is small and can be easily described, in reality,the “state” of a rig is determined by dozens of interacting activitiesand behaviors. Complete enumeration of all of the possible combinationsof activities and learning the probabilities of transitioning betweenall the states is typically intractable.

Instead, this invention makes use of an alternative conceptualization of“state” which is shown in FIG. 2. Here, each activity is represented asa column and time progresses from top to bottom. In this embodiment,each activity is represented as a binary variable (grouped binaryvariables are mutually exclusive, e.g., it is impossible to be drillingand pulling-out-of-hole simultaneously) which is updated at regularintervals (the intervals may be variable dependent, not shown). FIG. 2shows a state evolution where the rig was initially drilling, thenstopped to pull out of hole, and finally reached the state “out ofhole.” Meanwhile, the volume in the pits was varied as the pits wereemptied, and barite was added to the pits on and off during pulling outof the hole. Also, the iron roughneck was engaged and disengaged severaltimes during the process (to disconnect pipe stands, for example).

As shown in the embodiment of FIG. 2, the state may consist entirely ofbinary valued variables, but a state may also include discrete variablestaking multiple possible values (e.g., the number of pipe-stands in thewell), as well as real-valued states (e.g., hole depth).

Estimating rig state as a number of discrete variables has a number ofbenefits over classical state estimation and tracking, for example, toaccount for all the various rig behaviors in a classical system requiresan exponentially increasing number of discrete states, and an evenlarger number of inter-state transition probabilities.

Depending on the state specification, different algorithms making use ofdifferent data sources are implemented to detect different relativevariables. These algorithms use features from the relevant sensor data,together with machine learning algorithms for decision making. Forexample, to determine whether the rig is “pulling out of hole” analgorithm could utilize information from:

-   -   1. the recent changes in bit depth and current hole depth,    -   2. video of the pipe-stand, and pipe tracking outputs, and/or    -   3. recent or ongoing roughneck engaged/disengaged measures.

Information from each of these sensing modalities may be extracted usingfeature extraction approaches to generate low-dimensional, informationbearing representations of the data. Low-level or quickly changingstates that are likely to occur in a repeated sequence can be furtheraggregated into temporal models of likely behaviors within each largerstate. For example, video observations of the rig floor during the“pulling out of hole” state are likely to iteratively determine threestates in succession as depicted in FIG. 3.

Explicit small-scale state-transition models, like the one shown in FIG.3, are used to aggregate information temporally to improve evidence ofthe larger state inference (e.g., spending time in the activities shownin FIG. 3 adds credence to the “Pulling Out Of Hole” state).

Throughout processing, each video camera 102 may incorporate humandetection and tracking processing to identify the locations of people onthe rig, flag any important occlusions (e.g., someone is blocking thecamera's view of the drill hole), and/or record a person's motion andactivities. These activities may then be utilized in automated T&Mreporting.

Information from the estimated rig state is also provided to systems forthe identification of uncommon or dangerous state transitions andautomated rig control systems (discussed below). Information about theglobal rig state may then be directly utilized in improving automatedalarm generation systems. For example, information that there is bariteadded to the pits is used to change the influx detection system toignore pit volume measurements for the time being. Similarly,information about the states “pipe static” and/or “pumps off” indicatesthat any positive change in flow-out and pit-volume may be highlyindicative of down-well influx, since no other activities should beinfluencing those measurements.

In addition to altering real-time processing and generally improving rigoperations, computer vision based rig state detection and personneltracking may also be used to automatically annotate video data as it iscollected, along with the rig state, number of persons in scene, andother relevant sensor information. This processing may automatically addadditional information to be attached to the video stream, to enablefast searching for discrete events (e.g., “show me every instance wheretripping out of hole took more than 20 minutes but hole depth was lessthan 2000 ft”), which is currently intractable and extremelytime-consuming with existing software. Embodiments of the system mayshow a prototype state visualization tool and sample frames from each“pipe-pulled” event found in the video. Each of these frames may providea direct link to the corresponding point in the video.

Each state-vector is recorded as a function of time and/or as part of arelational database (or other suitable software, e.g., a spreadsheet).These state-vector data sets are then used to automatically generatereports, and are made available to T&M analysts after data is collected.These data sets enable automatic searching over massive amounts of videothat has been automatically tagged with T&M relevant information. Forexample, this enables identification of all events of type Y that tookmore than M minutes across all wells and rigs, using only alreadycollected video and sensor data. This represents a large time savingsand increase in the power and efficiency of T&M reporting.

Each camera 102 may also keep track of all people in the scene usingautomated person-detection algorithms. When multiple cameras are viewingthe same region, person locations can be estimated and provided as partof the automated T&M reporting and database—e.g., “Person 1 detected atlocation (x1,y1), Person 2 detected at location (x2,y2).” Persons may beautomatically anonymized by blurring of the pixels containing detectedpersons for privacy and reporting reasons.

Since the joint computer vision/down-well signal processing approachesdescribed provides a state descriptor vector, which can be of very highdimension, estimating complete inter-state transition probabilities isintractable. However, by aggregating states into larger-picture states,or considering small sub-sets of states only (e.g., only the statedescriptors shown in the left most group of FIG. 2) it is possible toaccurately enumerate likely and dangerous transition probabilities byincorporating a priori expert information about realistic statetransitions and state transitions that should be rare or impossible(e.g., a transition from “drilling” directly to “out of hole” mostlikely indicates a sensor failure or error, other transitions mayindicate dangerous or environmentally unsafe behaviors or situations).

Information from the computer vision and additional sensor stateestimation techniques may also be used to determine appropriate rigcontrol actions and automate rig behaviors and other processes through asupervisory control and data acquisition (“SCADA”) control system.

FIG. 4 shows one embodiment of the disclosed system in which multiplecameras 102 may be used to monitor various aspects of the rig state. Thecameras 102 are operably connected to a processor 110. In thisparticular embodiment, the processor 110 is operably connected to adatabase 112, alarm 114, machinery control system 116 and sensor 120. Inrelated embodiments one, some or all of these devices may be operablyconnected to the processor 110. Additionally, other embodiments mayinclude, for example, multiple sensors 120.

FIG. 5 shows the steps of a method for estimating rig state. Thedisclosed method comprises the steps of sensing aspects of rig state302, collecting visual data, 304, processing visual data to determineperson location 306, compiling multiple sources of rig data 308,estimating rig state 310, refining sensor data 312, causing orinhibiting automated activities 314, alerting staff 316, annotatingvisual data 318, recording data 320 and comparing compiled data againstpreviously recorded data 322. Other embodiments may include some or allof these steps in any sequence.

Disclosed embodiments relate to a system for estimating global rigstate. The system may include at least one camera 102 operably connectedto at least one processor 110, wherein the camera is capable ofgathering visual data regarding at least one variable of rig state. Theprocessor 110 is also capable of compiling rig state data, estimatingglobal rig state, or both. The system may also include at least onesensor 120 for measuring at least one variable related to global rigstate wherein the sensor 120 is operably connected to the processor 110.The system may additionally include a model incorporating multiplevariables related to rig state.

In certain embodiments, the model variables can be updated in real time,the compiled data may be displayed to a user, and/or the estimated rigstate may be used to refine data collected from said sensors.

Some disclosed embodiments may also include a database 112 operablyconnected to the processor 110, wherein the processor 110 is capable ofcomparing current data against historical data in the database.

Additional embodiments may include an alarm system 114 for alertingstaff to the occurrence of a pre-determined condition and/or a machinerycontrol system 116 to cause or inhibit certain automated activities.

In some embodiments, the visual data, sensor 120 measurements, estimatedrig state or any combination thereof are searchable by the processor110.

Some disclosed embodiments relate to a method for estimating rig state.The method may comprise the steps of sensing at least one aspects of therig state 302 using at least one sensor 120, collecting visual data 304corresponding with the sensor data using at least one camera 102,compiling multiple sources of rig data 308 and estimating the overallrig state 310.

In certain embodiments, the estimated overall rig state may be used torefine the gathered sensor data 312 and/or the determined personlocation may be used to cause or inhibit certain automated activities314.

Some embodiments may also include the steps of processing visual data todetermine person location 306, alerting staff 316 to the occurrence ofpredetermined conditions, annotating gathered visual data 318 withcorresponding rig state data, recording the compiled data 320 for futurereference and/or comparing the compiled data against a database ofpreviously recorded data 322.

What is claimed is:
 1. A system for estimating global rig statecomprising: at least one camera operably connected to at least oneprocessor, wherein said camera is capable of gathering visual dataregarding at least one variable of rig state and said processor iscapable of compiling rig state data, estimating global rig state, orboth; at least one sensor for measuring at least one variable related toglobal rig state wherein said sensor is operably connected to saidprocessor; and a model incorporating multiple variables related to rigstate.
 2. The system of claim 1, wherein the model variables can beupdated in real time.
 3. The system of claim 1, wherein the compileddata is displayed to a user.
 4. The system of claim 1, wherein theestimated rig state is used to refine data collected from said sensors.5. The system of claim 1, further comprising a database operablyconnected to the processor, wherein said processor is capable ofcomparing current data against historical data in the database.
 6. Thesystem of claim 1, further comprising an alarm system for alerting staffto the occurrence of a pre-determined condition.
 7. The system of claim1, further comprising a machinery control system to cause or inhibitcertain automated activities.
 8. The system of claim 1, wherein thevisual data, sensor measurements, estimated rig state or any combinationthereof are computer searchable.
 9. A method for estimating rig statecomprising: sensing various aspects of the rig state, collecting visualdata corresponding with said sensor data, compiling multiple sources ofrig data, and estimating the overall rig state.
 10. The method of claim9, wherein the estimated overall rig state is used to refine thegathered sensor data.
 11. The method of claim 9, further comprisingprocessing visual data to determine person location.
 12. The method ofclaim 11, wherein the determined person location is used to cause orinhibit certain automated activities.
 13. The method of claim 9, furthercomprising alerting staff to predetermined conditions.
 14. The method ofclaim 9, further comprising annotating gathered visual data withcorresponding rig state data.
 15. The method of claim 9, furthercomprising recording the compiled data for future reference.
 16. Themethod of claim 9, further comprising comparing the compiled dataagainst a database of previously recorded data.